♦ 



Express Mail Label No: EL449701 129US 



Date of Deposit: April 28, 2001 



fey 



POST OFFICE ADDRESS: 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

PATENT APPLICATION 

APPLICANTS/INVENTORS : Bharti Temkin, Ransom Canyon, TX 

a U.S. citizen; 



Kirk Watson, Round Rock, TX 
a U.S. citizen; and 

Eric Acosta, Lubbock, TX, a U.S. citizen 

c/o Texas Tech University 
Transfer and Intellectual Property 
P.O. Box 42007 
Lubbock, Texas 79409-2007 

DEVELOPMENT OF STEREOSCOPIC-HAPTIC 
VIRTUAL ENVIRONMENTS 

Texas Tech University, a not-for-profit 
institution 

Perkins, Smith & Cohen, LLP 
Jacob N. Erlich, (Reg. No. 24,338) 
Jerry Cohen (Reg. No. 20,522), 
Stephen Y. Chow (Reg. No. 31,338) 
Harvey Kaye (Reg. No. 18,978) 
Christine M. Kuta (Reg. No. 38,001) 
One Beacon Street 
Boston, MA 02108 

(617) 854-4000 voice and fax 617-854-4040 

Correspondence to Mr. Cohen 

617-854-4282 direct line 

E: jerry_cohen@pscboston.com 



INVENTION TITLE: 



ASSIGNEE: 



ATTORNEYS/AGENTS: 



To: 



Sir: 



Honorable Commissioner of Patents and Trademarks 
Washington, D.C. 20231 



Your applicants, named above, hereby requests acceptance of the enclosed 
regular utility application entitled as stated above pursuant to 35 U.S.C. § 1 1 1(a). 



5 



DEVELOPMENT OF STEREOSCOPIC-HAPTIC VIRTUAL ENVIRONMENTS 
FIELD AND BACKGROUND OF THE INVENTION 



integrating haptic or touching and stereoscopic or three dimensional visual computer 
5 interfaces, and is more particularly associated with such apparatuses and software as applied 
to surgical simulations. The integration in computer generated virtual environments of 
interfacing of physically felt force-feedback devices (haptic interface devices) that provide 
the touch or feel sensation and stereoscopic visual interfaces that provide visual three 
dimensional (3D) objects have not been extensively developed. Those that exist use 
10 expensive complex and often ad hoc hardware and software that are difficult to implement 
and more difficult to service and/or modify. High end, expensive graphic workstations, e.g. 
Sun Microsystems with specialized hardware and/or software have been so used, but are not 
amendable to routine use due to the complexity and the expense. 



fundamentally from other human sensory modalities in that it is bilateral in nature: to touch 
an object, the object must "push" back. In computer generated haptics, a computer interface 
device is used that provides a physical touch to the human that corresponds to the real 
three-dimensional sense of touch that allows one to feel textures and shapes of objects, 

2 0 modulate objects and even deform objects. 

Two major components of computer haptics are collision detection of virtual objects 
with the haptic interface device, and the determination and application of a force feedback to 
the user via the haptic interface device. Prior art data structures and algorithms applied to 
haptic rendering have been adopted from non-pliable surface-based (two dimensional) 

25 graphic systems. These prior art techniques and systems are inappropriate and limited due 




The present invention relates to apparatus and software cooperatively 
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Before proceeding some definitions will be helpful in understanding the invention. 
HAPTICS: Haptics refers to touch. Human sense of touch, human haptics, differs 



2 



to the different characteristics required for haptic rendering of three dimensional or 
volumetric soft bodies. 

STEREOSCOPIC DISPLAY: Stereoscopic herein refers to true 3D viewing of 
objects. In many prior art systems 3D images are rendered for viewing in a 2D format (e.g., 
on a flat monitor). True 3D viewing provides the user with the appearance of true depth. 

When viewing an object in 3D, both eyes see the object from a slightly different 
perspective. Each eye perceives a slightly different image, with the brain naturally fusing 
these two images into one by matching up the similarities and adding in the small 
difference. The result is the illusion of depth. This process of stereopsis can be simulated 
by 1) generating the two dimensional image for each eye, and 2) presenting the left eye 
image only to the left eye and the right eye image only to the right eye. This brief review 
suffices since stereoscopy is well understood in the art. 

Such prior art devices are even more limited if applied to teaching the complex skills 
associated with medical surgery. Surgery skills are taught on live patients or animals. A 
computer system that provides true touch and corresponding 3D vision for volumetric soft 
bodies is needed in the art. 

It is an object of the present invention to generate a stereoscopic-haptic computer 
interface suitable for providing virtual environment. 

It is another object of the present invention to result in the illusion of being able to 
"see" and "touch and feel" 3D virtual environment, for example, a surgeon operating on a 
virtual patient, but with true-to-life touching and seeing. 

It is yet another object of the present invention to produce cost-effective, real-time, 
interactive, stereoscopic haptic virtual environment with minimal hardware. 

It is yet another object of the present invention to transform the virtual environment 
from a flat, formless 2D experience to a more realistic and solid 3D experience. 



It is still another object of the present invention to provide 3D organ models for 
study by and training of students. 

SUMMARY OF THE INVENTION 
The objects set forth above as well as further and other objects and advantages of the 
5 present invention are achieved by the embodiments of the invention described herein below. 

The present invention includes generating a stereoscopic-haptic interface to simulate 
human sense of touch and depth perception with display 3D information in a flat 2D 
environment. The haptic interface and stereoscopic technologies include: 1) generating 
stereoscopic images using, active stereoscopic displays where shutter glasses shutter on and 
10 off in synchrony, interlaced video displays and a stencil buffer to draw images on different 
_ scan lines; and 2) making a stereoscopic-haptic system using a point-based haptic device, all 
y3 with supporting software. The haptic environments are created independently of their 
=C graphical representations allowing for independent development and troubleshooting, but 
B where the haptic and visual presentations are coordinated. The visual 3D is accomplished 
H 15 with a single camera that is moved to simulate the position of the left eye and then the right 
□ eye. The image in each position is stored in a stencil buffer that records the left eye image 

rij on even scan lines and the right eye image on the odd scan lines of an interlaced video 

q system. The presentation of the two images via the shuttered glasses to the corresponding 

La. 

eye in a coordinated fashion and the coordinated haptic device allows for 3D visualization 
2 0 and corresponding touching to be experienced by the user in the virtual environment. 

An advantage of the present invention is the operation without costly extra hardware 
and software as described above. The present invention uses the vertical synchronization 
signals directly from the video signal to switch the shutters on the glasses and the 
multiplexing of the left and right images by using interlaced scan lines minimizes complex 
2 5 and costly hardware and synchronizing software. The result is that the haptic and the 3D 
visual are combined with a minimum of extra hardware and software to provide a cost 
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effective virtual environment suitable for teaching and practicing complex, difficult 
techniques that depend on sight and touch. 

For a better understanding of the present invention, together with other and further 
objects thereof, reference is made to the accompanying drawings and detailed description. 

BRIEF DESCRIPTION OF THE DRAWING 

Figure 1 is a flow diagram implementing a haptic application and interface device; 
Figure 2 is a representation of left and right eye images combined to form 3D 
stereoscopic image; 

Figure 3 represents parallax issues in 3D presentations; 
Figure 4 is a parallel camera model; 

Figure 5 is a flowchart of a preferred embodiment process architecture; 
Figure 6 shows a scene with the stencil rendered, and 

Figure 7 is a diagram of showing the independence of the haptic and visual 
representation. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

Figure 1 is a flow diagram of implementing a programmable haptic pointing device 
in an environment that simulates human touch. The location, 3D shape, and feel of the object 
are programmed so that, when the user moves the interface device, the positional information 
in virtual space is retained, and based on what the object being touched (as programmed) the 
device provides a force feedback to the hand of the user. 

Referring to Figure 1, the scene is initialized 2 as to the touch and shape of the 
objects in the scene and their virtual positions. The relative virtual position 4 of the haptic 
interface is determined. As the interface device is moved, the touching of the objects in the 
scene is determined by resolving collisions 6 with the virtual objects. Depending on the type 
of object encountered the interface device presents back 10 to the user a force calculated 8 to 
duplicate the force that an actual object would provide. 
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The force interaction of Figure 1 provides means for training of cognitive skills to 
users, but can also allow evaluations of sensory-motor skills; for example, an advanced 
haptic surgical simulation not only is capable of teaching the complex physical interaction 
skills, but, as mentioned above, also can record and play back the mechanical signals. This 
5 later feature can be used to monitor trainee's skill acquisition, customize the training 
program, or even let a trainee practice on prerecorded actions of an expert. 

Figure 2 is a diagram showing the perspective binocular disparity between the left 
eye 12 and the right eye 14 when viewing a scene and the scene frontally shown. 16. This 
disparity provides the human brain 18 with information that is the stereoscopic 3 D effect. 
10 Figure 3 shows the generating of left/right eye images. The depth cue that produces 

_ the stereoscopic effect is binocular disparity discussed above. However, it is possible for 
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J3 several depth cues to either reinforce 20 or conflict 22 with each other. The latter should be 
=F avoided when stereoscopic 3D virtual images are generated. 

ffl Figure 4 shows a parallel camera model 24 that is used in a preferred embodiment 

M> 15 of the present invention. Nearly all computer displays are flat. Thus, the images projected on 
D these displays are also flat. However, distortion can occur as we attempt to simulate the 

human eye, since the eye has a curved surface. The simple parallel camera model 
substantially removes distortion in projection, but there are 1) overlapping of the viewing 
spaces of the cameras is not always complete, which can cause a fraction of the image to be 
20 not stereoscopic, and 2) viewer is forced to converge his own eyes, since the cameras do not 
converge. The viewer having to keep focus on the monitor can cause a disparity between the 
depth cues of convergence and accommodation (changing of the focal lens of eyes to focus 
on an object). With practice, however, this disparity is not a problem. 

In a preferred embodiment an active stereoscopic display is used for image 
25 presentation, where shutter glasses shutter on and off in synchrony, presenting one image at 
a time to each eye in quick succession while the other eye is covered. The switiching 
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presentation is performed at 60 Hz so that the user "sees" only one stereoscopic image. 
When rendering a complex, dynamic, real time scene the images must be switched quickly 
enough or the stereoscopic effect would be lost. This problem is solved in a preferred 
embodiment by using the interlaced mode available on many video cards and a stencil 
5 buffer. Interlaced video displays only the even set of horizontal scan lines on a vertical scan 
followed by the odd set of scan lines on the next pass. This allows for smoother animation 
with a relatively low refresh rate as both the left and right eye images are displayed 
simultaneously on sequential vertical scans seen as one image, and the stereoscopic effect 
does not suffer. There is a concurrent loss of resolution that is not a problem since the brain 
10 merges the images and the resulting resolution is still acceptable. The stencil buffer is used 
_ to render the left eye images only on the even scan lines (on stencil), and the right eye image 
J3 only on the odd scan lines (off stencil). This preferred embodiment minimizes the timing of 
=P switching the shutters on each vertical synch pulse from the display. The left eye image is 
ffl on the screen during one entire vertical scan of even lines and the right eye is on the screen 
M* 15 on the next vertical scan of odd lines. 

O The basic idea is to move the camera to the left and render the scene on stencil and 

fy then move the camera to the right and render off stencil. Then the right and the left eye 

□ images are interlaced together. When the video is placed in an interlaced mode, the result 

will be that the left eye image is displayed separately from the right eye image. The shutter 
2 0 glasses are automatically synchronized (as known in the art via a transmitter plugged into 
the output of the video card) to the vertical sync signals of the video card when in interlaced 
mode, the left eye will only see the left eye image and the right eye will only see the right 
eye image. Figure 5 shows a depiction of the scene of a turtle with the stencil on. Here, for 
example, only the even horizontal scan line of the interlaced video are present that are to be 
2 5 presented to the left eye in the example described herein. The right eye will be presented 



7 



with the odd horizontal scan lines coordinated by the video vertical retrace, synchronization 
or vertical blank signal associated. 

Referring to Figure 6, the system is initialized 26 including a "callback" function 
that is used by the programmer to graphically render the haptic scene and ensure that the 
optical and haptic presentations and operations are coordinated, consistent and synchronized 
to each other. This task is well within the capabilities of those skilled in the art. The stencil 
and shape information is installed to accommodate the haptic and the two camera visual 
environment. The scene is 3D visually processed by moving the camera and turning the 
stencil on and off to create the visual 3D effect. The visual is then incorporated with the 
haptic in the main loop 26. Figure 7 indicates the independence of the 3D visual and the 
haptic interfaces. However, the combination is coordinated as described herein to present a 
realistic virtual scene with touch and sight. Although not discussed adding coordinated 
audio would enhance the present invention in particular applciaitons. Such an addition and 
coordination are within the skill of the art when the present haptic/visual invention is 
disclosed. 

The above techniques were applied in a preferred embodiment to an existing 
non-stereoscopic haptic scene. This haptic scene consists of several simple shapes that can 
be moved, rotated, translated and scaled. By applying the technique to the rendering 
function, it is possible for the user to view the scene stereoscopically using shutter glasses. 
The user also has the ability to dynamically adjust the perceived depth of the scene, and to 
control whether the objects appear to recess into the monitor or pop out of it. The user can 
view and feel the objects in three dimensions in real time. A simple stereoscopic haptic 
environment is finally created. 

A preferred embodiment of the present invention included a 300 MHz personal 
computer running the NT operating system. The video was run at 43 Hz interlaced with 
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shutter glasses. Virtually any computing system, operating system and shutter glasses, 
known in the art, can be used in preferred embodiments of the present invention. 

Although the invention has been described with respect to various embodiments, it 
should be realized this invention is also capable of a wide variety of further and other 
embodiments within the spirit and scope of the invention. 



